Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

🚇 Speedup PR benchmark #785

Merged
merged 2 commits into from
Aug 22, 2021
Merged

🚇 Speedup PR benchmark #785

merged 2 commits into from
Aug 22, 2021

Conversation

s-weigand
Copy link
Member

Running benchmarks on PRs takes forever to get feedback (~1h e.g. here) since it benchmarks each commit in between.
The most important information are the benchmark of the last release and the current commit before merging.
Thus we can omit running the benchmarks for all steps in between.
If more insight is needed at which specific commit in a PR a performance regression occurred it can always be run locally.

Change summary

  • Only run base compare commit (v0.4.0) and last HEAD commit on PR benchmark

Checklist

  • ✔️ Passing the tests (mandatory for all PR's)

@s-weigand s-weigand requested a review from a team as a code owner August 22, 2021 08:21
@github-actions
Copy link
Contributor

Binder 👈 Launch a binder notebook on branch s-weigand/pyglotaran/speedup-PR-benchmark

@s-weigand s-weigand changed the title Speedup PR benchmark 🚇 Speedup PR benchmark Aug 22, 2021
@codecov
Copy link

codecov bot commented Aug 22, 2021 •
edited
Loading

Codecov Report

Merging #785 (7b3a571) into staging (b960a98) will not change coverage.
The diff coverage is n/a.

Impacted file tree graph

@@           Coverage Diff           @@
##           staging    #785   +/-   ##
=======================================
  Coverage     83.9%   83.9%           
=======================================
  Files           75      75           
  Lines         4185    4185           
  Branches       752     752           
=======================================
  Hits          3512    3512           
  Misses         538     538           
  Partials       135     135

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b960a98...7b3a571. Read the comment docs.

@s-weigand
Copy link
Member Author

The benchmarking time is down to ~3min 🎉

@s-weigand
👌 Update comment w/o deleting the last one
7b3a571 
This way we a comment edit history
@sonarqubecloud
Copy link

Kudos, SonarCloud Quality Gate passed!    Quality Gate passed

Bug A 0 Bugs
Vulnerability A 0 Vulnerabilities
Security Hotspot A 0 Security Hotspots
Code Smell A 0 Code Smells

No Coverage information No Coverage information
0.0% 0.0% Duplication

@github-actions
Copy link
Contributor

Benchmark is done. Checkout the benchmark result page.
Benchmark differences below 5% might be due to CI noise.

Benchmark diff

Parametrized benchmark signatures:

BenchmarkOptimize.time_optimize(index_dependent, grouped, weight)

All benchmarks:

       before           after         ratio
     [dc00e6da]       [7b3a5712]
     <v0.4.0>                   
-        62.4±2ms       43.9±0.6ms     0.70  BenchmarkOptimize.time_optimize(False, False, False)
-         363±3ms         57.7±1ms     0.16  BenchmarkOptimize.time_optimize(False, False, True)
-        91.5±2ms       74.0±0.7ms     0.81  BenchmarkOptimize.time_optimize(False, True, False)
         90.8±2ms        86.3±30ms     0.95  BenchmarkOptimize.time_optimize(False, True, True)
         63.7±1ms       59.0±0.5ms     0.93  BenchmarkOptimize.time_optimize(True, False, False)
-         362±6ms        66.7±30ms     0.18  BenchmarkOptimize.time_optimize(True, False, True)
         89.5±1ms         92.2±2ms     1.03  BenchmarkOptimize.time_optimize(True, True, False)
+        92.9±2ms         112±40ms     1.21  BenchmarkOptimize.time_optimize(True, True, True)
             182M             181M     0.99  IntegrationTwoDatasets.peakmem_create_result
             200M             199M     1.00  IntegrationTwoDatasets.peakmem_optimize
-         280±6ms          224±4ms     0.80  IntegrationTwoDatasets.time_create_result
-      5.85±0.02s       1.44±0.03s     0.25  IntegrationTwoDatasets.time_optimize

jsnel
jsnel approved these changes Aug 22, 2021
View reviewed changes
Copy link
Member

@jsnel jsnel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍 🕐++

@jsnel jsnel merged commit 5e9eb22 into glotaran:staging Aug 22, 2021
@s-weigand s-weigand deleted the speedup-PR-benchmark branch August 22, 2021 10:21
@jsnel jsnel mentioned this pull request Sep 16, 2021
Merged
5 tasks
jsnel pushed a commit that referenced this pull request Sep 16, 2021
@s-weigand @jsnel
🚇 Speedup PR benchmark (#785)
3affa01 
* 🚇 Only run base compare commit and last HEAD commit on PR benchmark

* 👌 Update comment w/o deleting the last one (comment should have an 'edit' history)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@jsnel jsnel jsnel approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@s-weigand @jsnel